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EXECUTIVE  SUMMARY 


“insights  into  tomorrow ” 


i 


Part  of  our  College  mission  is  distribution  of  the 
students’  problem  solving  products  to  DoD 
sponsors  and  other  interested  agencies  to 
enhance  insight  into  contemporary,  defense 
related  issues.  While  the  College  has  accepted  this 
product  as  meeting  academic  requirements  for 
graduation,  the  views  and  opinions  expressed  or 
implied  are  solely  those  of  the  author  and  should 
not  be  construed  as  carrying  official  sanction. 


REPORT  NUMBER  88-0310 

AUTHOR(S)  major  peter  a.  blatchley,  usaf 

TITLE  AC-130U  GUNSHIP  WARRANTY  ANALYSIS:  LESSONS  LEARNED 

I.  Purpose :  To  analyze  the  methodology  used  by  the  AC-130U  Gunship 
Program  Office  to  evaluate  its  weapon  system  warranty  during  source 
selection,  and  to  draw  pertinent  lessons  learned  from  that 
experience . 

II.  Background:  DoD  is  now  required  by  law  to  procure  cost- 
effective  weapon  system  warranties  in  conjunction  with  most  major 
weapon  system  acquisitions.  But  few  useful  tools  are  available  to 
aid  in  the  analysis  of  these  warranties.  Lacking  a  useful  "off-the- 
shelf”  warranty  analysis  tool,  the  AC-130U  Gunship  Program  Office 
crafted  a  warranty  cost-benefit  analysis  methodology  and  used  it 
during  the  source  selection  of  the  Gunship.  The  challenges  faced  in 
the  development  and  implementation  of  the  AC-130U  warranty  analysis 
methodology  suggest  important  "lessons  learned"  for  future  source 
selections . 

III.  Discussion :  Recent  legislation  requiring  the  procurement  of 
cost-effective  warranties  on  most  major  weapon  systems  established 
the  requirement  for  warranty  cost-benefit  analysis  during  source 
selection  for  that  weapon.  Following  a  futile  search  for  a  useful, 
"off-the-shelf"  warranty  evaluation  tool,  the  AC-130U  Gunship 
Program  Office,  in  what  turned  out  to  be  a  pioneering  effort, 


CONTINUED 


developed  its  own  warranty  cost-benefit  analysis  methodology.  The 
strengths  of  this  in-house  effort  include  the  analysis'  grounding  in 
life  cycle  cost  theory,  involvement  of  functional  area  experts  in 
its  development  and  validation,  and  the  foresight  with  which  it  was 
crafted.  Its  shortcomings  include  its  failure  to  capture  many  of 
the  relevant  costs  associated  with  the  weapon  system  warranty, 
artificially-constraining  assumptions,  and  a  reliance  on  ultimately 
unobtainable  data. 

IV.  Conclusions :  DoD  is  required  by  law  to  procure  cost-effective 
warranties  on  its  major  weapon  systems.  Consequently  acquisition 
agencies  must  conduct  some  form  of  cost-benefit  analysis  to  ensure 
the  procurement  of  cost-beneficial  warranties.  Unfortunately  few 
useful  warranty  evaluation  tools  exist  today.  The  AC-130U  Gunship 
warranty  analysis  effort  highlights  many  of  the  shortcomings  that 
characterize  the  few  existing  tools:  a  requirement  for  non-existent 
data,  the  inherent  difficulty  in  forecasting  future  costs,  the 
difficulty  of  translating  physical  performance  measures  into  terms 
of  dollars  and  cents,  and  the  specialized  nature  of  available  tools. 

V.  Recommendation :  DoD  must  develop  easy  to  use,  off-the-shelf 
warranty  analysis  tools  for  program  office  use  during  the  critical 
source  selection  process.  The  tools  must  overcome  the  shortcomings 
of  the  few  existing  tools  as  cited  above. 


Chapter  One 


INTRODUCTION 

DoD  weapon  system  acquisition  is  one  of  the  elemental 
processes  that  underlie  the  nation's  ability  to  effectively 
apply  military  power  in  the  pursuit  of  national  interests 
worldwide.  How  well  we  acquire  effective  and  efficient  weapons 
— from  rifles  to  main  battle  tanks,  life  rafts  to  nuclear 
submarines,  radios  to  strategic  bombers--in  large  part  will 
determine  the  effectiveness  of  US  armed  forces  in  the  future. 

DoD  acquisition  agencies  have  been  the  target  of 
considerable  criticism  in  recent  years  as  a  result  of  a 
perception  that  these  organizations  have  not  been  providing  the 
services  weapons  of  the  right  kind,  quantity  or  quality. 

Congress  has  enacted  legislation  intended  to  improve  the 
acquisition  process.  Two  enactments  in  particular  have  had  a 
significant  impact  on  the  weapon  system  acquisition  process.  In 
the  Weapon  System  Warranty  Act  and  the  Defense  Procurement 
Reform  Act,  Congress  has  required  the  DoD  to  obtain  cost- 
effective  manufacturer  warranties  on  all  new  major  weapon 
systems  (13:5-9).  The  impact  of  this  legislation  has  been  a 
significant  change  in  how  the  services  acquire  weapons. 

While  the  services  had  procured  some  warranted  systems  prior 
to  these  two  enactments,  the  use  of  warranties  had  not  been 
widespread.  But  now  that  the  Weapon  System  Warranty  and  Defense 
Procurement  Reform  Acts  mandated  the  procurement  of  cost- 
effective  warranties  on  all  major  weapon  systems,  major  new 
decisions  were  required  of  acquisition  agencies.  The 
determination  of  whether  or  not  a  contractor's  proposed  weapon 
system  warranty  was  cost-effective  became  a  crucial  judgement. 
Besides  the  obvious  legal  requirement,  obtaining  a  cost 
effective  warranty  can  reduce  the  life  cycle  cost  of  owning, 
supporting,  and  using  a  new  weapon  system,  and  improve  its 
effectiveness. 

This  study  examines  the  process  employed  by  the  AC-130U 
Gunship  Program  Office  and  Source  Selection  Team  in  evaluating 
the  cost-effectiveness  of  the  warranties  proposed  in  three 
competitive  bids.  After  a  brief  examination  of  the  history  of 
weapon  system  warranties  within  DoD  (Chapter  2) ,  this  study 
presents  a  discussion  of  the  basic  theory  that  underlies 


warranty  evaluation  and  samples  some  typical  warranty  cost- 
benefit  analysis  methodologies  (Chapter  3) .  Chapter  4 
describes  the  Gunship  warranty  provisions  and  the  methodology 
used  to  evaluate  the  warranties  proposed  by  the  three  defense 
contractors  who  competed  for  the  nearly  $1  billion  contract. 
Chapter  5  offers  an  analysis  of  the  strengths  and  weaknesses  of 
the  Gunship  warranty  cost-benefit  analysis  effort.  Finally, 
Chapter  6  presents  conclusions  and  a  recommendation  for  the 
future  application  of  warranty  analysis  tools. 

The  goal  of  this  research  project  was  to  answer  the 
question,  "Are  there  lessons  to  be  learned  from  the  AC-130U 
Gunship  warranty  evaluation  effort  that  may  serve  to  improve 
future  warranty  cost-benefit  analyses?"  Such  a  question  would 
suggest  the  author  enters  the  research  with  knowledge  of  certain 
imperfections  within  the  Gunship  cost-benefit  analysis.  That  is 
indeed  the  case.  But  any  criticism  of  the  methodology  is,  in 
part,  self-criticism  as  the  author  was  one  of  the  prime 
architects  and  users  of  the  Gunship  warranty  evaluation 
methodology . 


Chapter  Two 


HISTORY  OF  WARRANTIES  AND  WARRANTY  COST  BENEFIT 

ANALYSIS  IN  DOD 


POD  WARRANTY  HISTORY 

Warranties  on  DoD  weapon  systems  are  a  relatively  new 
phenomenon,  although  they  were  clearly  in  use  prior  to  1984  when 
Congress  mandated  their  use.  As  early  as  1965,  the  Armed 
Services  Procurement  Regulation  (ASPR)  called  for  the  selective 
use  of  warranties  (14:14).  However,  the  ASPR  guidance  was 
widely  "...  interpreted  to  mean  that  use  of  an  extensive, 
long-term  warranty  should  be  the  exception  rather  than  the 
rule"  (3:2-2).  In  1978  "  .  .  .the  Air  Force  Systems  Command 
directed  the  expanded  use  of  warranties  in  Air  Force 
procurements"  (14:16).  That  same  year  saw  the  birth  of  the  Air 
Force  Product  Performance  Agreement  Center  (PPAC)  whose  mission 
was  "...  to  assist  Air  Force  activities  involved  in  the 
acquisition  of  defense  systems  ...  in  selecting,  structurina, 
pricing,  negotiating,  and  implementing  effective  Product 
Performance  Agreements,"  e.g.,  warranties  (7:2-1).  Beginnina  in 
1981,  the  DoD  Acquisition  Improvement  Program  focussed  attention 
on  the  use  of  warranties  to  improve  the  reliability  and 
supportability  of  weapon  systems  (14:16).  The  Federal 
Acquisition  Regulation  (FAR) ,  the  successor  to  the  ASPR  and 
Defense  Acquisition  Regulation  (DAR) ,  addressed  criteria  and  the 
government's  rights  in  the  structuring  and  use  of  warranties  in 
1984  (4:--).  In  the  years  just  before  the  Weapon  System 
Warranty  Act  and  the  Defense  Procurement  Reform  Act,  the 
acquisition  community  regarded  warranties  as  a  "standard 
option",  though  one  only  selectively  applied  (3:2-2).  Clearly 
the  history  of  warranties  in  the  DoD  predates  the  legislation 
that  now  requires  their  use. 

In  1984,  notwithstanding  DoD's  expanding  application  of 
weapon  system  warranties,  and  in  response  to  perceived  problems 
with  the  quality  of  certain  newly-acquired  weapons,  Congress 
passed  the  Weapon  Systems  Warranty  Act  (WSWA)  (Public  Law  98- 
212).  More  popularly  known  as  the  Andrews  Amendment,  the  Act 
was  added  to  the  DoD  Appropriation  Act  of  1984  as  Section  794 
(13:5-6).  The  WSWA  requires  warranties  on  all  new  weapon  system 
acquisitions  in  which  the  unit  cost  exceeds  $100,000,  or  the 


total  procurement  cost  exceeds  $10  million  (10:1).  The 
enactment  stipulates  that: 

[N]o  funds  may  be  obligated  or  expended  for  the 
procurement  of  a  weapon  system  unless  the  prime 
contractor  guarantees  that  the  system  is  designed  and 
manufactured  to  meet  the  government's  performance 
requirements  and  is  free  from  all  defects  in  material 
and  workmanship  that  would  cause  it  to  fail  to  conform 
to  performance  reqirements  (10:1). 

The  effect  of  this  legislation  was  weapon  system  warranties 
were  now  required  by  law,  and  in  only  two  circumstances  could 
the  Secretary  of  Defense  waive  that  requirement:  the  warranty 
would  not  be  in  the  interest  of  national  defense,  or  it  would 
not  be  cost  effective  (7:1-4). 

The  WSWA  represented  a  significant  departure  from  FAR 
procedures  Lor  government  acquisition  agencies  and  defense 
contractors  alike.  For  DoD,  warranties  were  no  longer  an  option 
available  to  the  Program  Manager  but  a  statutory  requirement. 
Perhaps  more  significantly,  the  WSWA  saddled  the  prime 
contractor  with  overall  responsibility  for  warranting  the 
quality  of  the  materials,  the  workmanship,  and  essential 
performance  of  the  weapon  at  the  system  level  (13:7-3). 
Previously,  the  FAR  focussed  on  the  component  level  in  its 
warranty  treatment  and  had  held  the  manufacturer  of  the  specific 
component  liable  for  defective  material  or  workmanship  (13:7-8). 
With  WSWA,  the  prime  contractor  was  now  responsible  for  making 
all  the  parts  and  the  system  itself  work  right. 

The  immediate  result  of  the  WSWA  was  a  storm  of  protest  and 
criticism  from  the  DoD  and  defense  contractors  alike.  Strong 
lobbies  were  mobilized  in  Congress,  attacking  the  Public  Law  on 
the  basis  of  its  certain  unfavorable  impact  on  the  cost  of  new 
weapon  systems  and  the  loss  of  flexibility  in  government  program 
management  (13:8).  The  following  year,  1985,  largely  in 
response  to  this  lobbying,  in  particular  the  commercial  lobby. 
Congress  modified  the  original  law  by  passing  the  Defense 
Procurement  Reform  Act  (Public  Law  98-525) ,  as  an  amendment  to 
the  DoD  Appropriations  Act  of  1985. 

The  Defense  Procurement  Reform  Act  sustained  the  basic 
requirements  of  its  predecessor.  Weapon  system  warranties  were 
still  required  on  materials,  workmanship,  and  performance,  and 
the  prime  contractor  was  still  responsible  and  liable  for  system 
level  performance.  The  significant  change  embodied  in  the 
latter  Public  Law  reflected  Congress'  intent  that  the  DoD 
procure  only  cost-beneficial  warranties.  Consequently,  the 
Defense  Procurement  Reform  Act  required  the  DoD  to  conduct  a 
cost-benefit  analysis  to  ensure  acquisition  agencies  procure 
only  warranties  economically  favorable  to  the  government.  The 


Act  ensured  Congressional  oversight  into  the  warranty 
procurement  process  as  it  required  the  notification  of  Congress 
in  the  event  an  acquisition  agency  proposed  to  procure  a  non- 
cost-beneficial  warranty  (16:3).  The  DoD  has  acknowledged  the 
Defense  Procurement  Reform  Act  by  modifying  its  Defense  Federal 
Acquisition  Regulation  Supplement  (DFARS) .  DFARS  Subpart 
46.770,  "Use  of  Warranties  in  Weapons  System  Procurements," 
encompasses  the  tenets  of  the  new  legislation  including  the 
requirement  for  a  cost-benefit  analysis  (4:46.7-3  to  7-8). 

The  use  of  warranties  in  DoD,  then,  has  progressed  from  an 
allowance  for  the  selective  implementation  of  component  level 
warranties  by  the  ASPR,  DAR,  and  FAR,  to  the  statutory 
requirement  for  the  procurement  of  cost-beneficial,  system-level 
warranties  on  major  weapon  systems.  But  to  what  extent  has  the 
DoD  performed  the  required  cost-benefit  analysis  in  the 
procurement  of  weapon  system  warranties? 


DoD  WARRANTY  COST-BENEFIT  ANALYSIS  HISTORY 

In  February  1937,  the  Government  Accounting  Office  (GAO) 
reported  that  in  the  97  procurement  contracts  reviewed,  in  only 
nine  had  "  .  .  .the  procurement  activities  performed  [warranty] 

analyses  that  addressed  basic  cost-effectiveness  criteria" 
(10:3).  The  GAO  determined  the  DoD  had  spent  S180  million  on 
warranties  for  which  there  had  been  "...  no  determination 
that  the  benefits  to  be  derived  under  the  warrant [ies]  were 
worth  the  cost"  (10:3).  What  are  the  reasons  behind  this 
apparent  failure  to  comply  with  the  intent  of  Congress  as 
reflected  in  the  Defense  Procurement  Reform  Act? 

Unfortunately  the  GAO  report  shed  little  light  on  any 
underlying  causes  of  DoD's  failure  to  perform  the  cost-benefit 
analyses  required  by  the  Defense  Procurement  Reform  Act.  But 
the  reasons  became  apparent  to  the  author  during  the  planning 
and  the  conduct  of  the  source  selection  for  the  AC-130U  Gunship 
aircraft  in  the  spring  of  1987. 

Primary  among  the  reasons  is  the  lack  of  guidance  available 
within  the  DoD  for  conducting  warranty  cost-benefit  analyses. 
Indeed,  in  the  months  leading  up  to  the  Gunship  source 
selection,  the  Program  Office  was  unable  to  obtain  anything 
resembling  a  useful  model  or  methodology  for  such  an  analysis. 
The  ultimate  futility  of  our  search  is  echoed  by  the  Defense 
Systems  Management  College  which  reported  in  its  Warranty 
Handbook  that  "...  [t]here  is  no  DoD  warranty  cost- 
estimation  model  that  addresses  all  the  warranty  requirements  of 
.  .  ."  the  Defense  Procurement  Reform  Act  (3:7-13). 

However,  several  DoD  warranty  cost  models  have  evolved,  some 
tailored  for  a  specific  type  of  warranty,  others  addressing  only 


a  subset  of  the  total  costs  involved.  A  sampling  of  the 
existing  warranty  evaluation  tools,  preceeded  by  a  discussion  of 
their  underlying  cost  estimation  methodology  is  presented  in 
Chapter  3. 


Chapter  Three 


EXISTING  WARRANTY  EVALUATION  TOOLS 


UNDERLYING  THEORY 

There  is  nothing  particularly  exotic  about  the  theory 
underlying  warranty  cost-benefit  analysis.  "In  analyzing 
warranties,  the  primary  concern  of  both  buyer  and  seller  is 
cost.  Accordingly,  most  of  the  models  of  warranty  process  that 
have  been  developed  are,  in  essence,  economic  in  nature"  (1:6- 
23).  In  other  words,  warranty  analysis,  from  either  the  buyer 
or  seller's  perspective,  involves  capturing  or  estimating  the 
costs  and  benefits  associated  with  the  execution  of  the 
warranty.  From  the  buyer's  or  government's  perspective,  the 
estimated  cost  must  be  compared  to  the  benefit  expected  to  be 
derived  from  the  warranty  in  making  the  decision  to  buy  the 
warranty. 

Indeed,  were  warranty  analysis  merely  a  matter  of  tallying 
list  of  costs  and  balancing  them  with  the  expected  benefits, 
then  the  task  of  warranty  evaluation  would  be  a  trivial  one. 

One  of  the  challenges  inherent  in  warranty  cost-benefit  analysi 
is  translating  into  dollars  seemingly  unquantif iable 
characteristics  of  a  weapon  system's  operational  and  support 
performance.  The  warranty  model  or  methodology  must  "... 
capture  [the]  essential  economic,  operational  and  statistical 
features.  These  will  vary  by  contract,  by  item,  by  use 
environment  and  doctrine,  and  by  a  host  of  other  factors  as 
well"  (1:6-24) .  The  task  quickly  takes  on  Herculean 
proportions.  As  a  result  of  the  incredible  complexity  involved 
in  warranty  analysis,  no  single,  universal  warranty  evaluation 
model  is  currently  available  (1:6-24).  Instead,  there  exist  as 
many  models  as  there  are  types  of  warranties  and  items 
warranted. 

In  summary: 

.  .  .  the  methods  available  for  estimating  warranty 
costs  are  not  really  any  different  than  those  used  for 
estimating  other  types  of  costs.  However,  the  real 
difference  in  the  warranty  area  is  that  there  is  a  great 
deal  of  emphasis  being  placed  on  estimating  costs  [when] 


the  data  base  required  for  this  is  seriously  lacking 

(16:18) . 

At  a  conceptual  level,  warranty  cost-benefit  analysis 
involves  the  comparison  of  benefits  and  costs  associated  with 
the  subject  warranty  in  an  effort  to  make  "  .  .  .the  best  choice 
from  a  number  of  warranty  alternatives"  (16:15).  (One 
alternative,  of  course,  is  choosing  not  to  buy  a  warranty  shown 
to  be  non-cost-beneficial.)  Consequently,  three  basic  methods 
of  estimating  warranty  costs  and  benefits  form  the  foundation  of 
the  majority  of  cost-benefit  analyses.  These  methods  are 
classic  cost  estimating  methodologies,  not  peculiar  to  warranty 
cost  evaluation,  and  include  the  "rule  of  thumb"  or  "heuristic" 
approach,  the  "cost-estimating  relationship"  (CER) ,  and  the 
"bottom-up  accounting  method"  (16:15). 

The  "rule  of  thumb"  or  "heuristic"  approach  to  the 
estimation  of  costs  in  general,  and  warranty  costs  in 
particular,  involves  establishing  the  relationship  between  one 
cost  element  and  another.  For  example,  one  rule  of  thumb 
commonly  used  to  estimate  the  cost  of  a  new  home  relates  the 
cost  to  the  living  space  measured  in  square  feet.  Ideally,  this 
approach  would  be  supported  by  historical  data,  but  often  a  rule 
of  thumb  is  based  only  on  experience  and  intuition. 

Consequently  a  rule  of  thumb  can  be  useful  in  the  face  of  a 
limited  or  non-existent  data  base.  However,  the  user  must 
remember  this  approach  will  yield  only  a  rough,  order  of 
magnitude  approximation  of  the  subject  cost  element  (11:74; 
16:16)  . 

The  second  methodology  available  for  the  estimation  of 
warranty  costs  and  benefits  is  the  Cost  Estimating  Relationship 
(CER) .  The  CER  is  a  statistically  based  parametric  relationship 
between  warranty  cost  and  some  other  variable  or  variables,  and 
consequently,  is  more  rigorous  and  potentially  accurate  than  the 
heuristic.  The  development  of  a  CER  is  a  complicated  and 
involved  process  that  depends  on  the  existence  of  an  extensive 
historical  data  base.  Once  developed  though,  a  CER  can  yield 
more  accurate  cost  estimates  than  those  generated  heuristically . 
One  other  significant  liability  associated  with  the  CER  also 
stems  from  its  grounding  in  historical  data.  A  CER  will  only  be 
valid  in  the  analysis  of  warranty  costs  for  systems  whose 
technology  is  similar  to  that  represented  in  the  historical  data 
base.  A  CER  cannot  be  applied  with  any  degree  of  fidelity  to 
systems  exploiting  technologies  radically  different  than  those 
captured  in  the  available  data  base  (11:74;  15:3-9,  4-6;  16:16- 
17)  . 

A  third  underlying  methodology  of  warranty  cost  estimation, 
the  bottom-up  accounting  method,  attempts  to  capture  every  cost 
element  that  comprises  the  total  cost  of  the  warranty.  Such  an 
approach  offers  the  potential  for  the  most  accurate  warranty 


cost  estimates.  It  relies  heavily  on  historical  data  as  well  as 
an  ability  to  accurately  forecast  future  costs.  Consequently, 
the  effective  use  of  a  warranty  model  based  on  a  bottom-up 
accounting  method  will  be  hampered  by  non-existent  historical 
data,  or  inaccurate  forecasts.  Further,  a  bottom-up  accounting 
methodology  that  captures  cost  elements  generated  via  heuristics 
or  CER  will  yield  forecasts  no  more  accurate  than  the  weakest 
estimate  it  includes  (11:74;  15:4-9;  16:17). 

In  summary,  the  preponderance  of  warranty  evaluation  tools 
available  today  capitalize  on  one  of  these  cost  estimating 
approaches.  As  indicated,  warranty  cost  estimation  is  not 
markedly  different  than  the  forecasting  of  other  types  of  costs. 
But  effective  warranty  cost  estimation  requires  types  of  data 
that  are  not  widely  available,  the  quantification  of  cost 
elements  that  defy  translation  to  dollars  and  cents,  and  the 
ability  to  accurately  forecast  future  costs.  Consequently, 
warranty  cost  estimation  is  difficult,  and  the  art  and  science 
of  warranty  evaluation  has  only  recently  received  the  attention 
that  will  ultimately  result  in  the  development  tools  useful  to 
the  acquisition  agencies  and  source  selection  activities. 


WARRANTY,.  EVALUATION  TOOLS 

The  following  warranty  evaluation  tools  typify  the 
methodologies  now  emerging  to  assist  the  program  office  in  its 
cost-benefit  analysis  of  weapon  system  warranties. 


U_S AF_R  I  W_  Model 

A  Reliability  Improvement  Warranty  (RIW)  is  a  specific  type 
of  warranty  "...  intended  to  provide  the  contractor  with  an 
incentive  to  improve  equipment  reliability  during  its 
operational  life"  (12:5).  Not  all  warranties,  not  even  all 
reliability  warranties  attempt  this.  Some  merely  warrant  the 
maintenance  of  a  specified  level  of  reliability  performance. 

But  the  Air  Force  became  an  early  proponent  of  this  particular 
type  of  warranty  when  commercial  air  carriers  attributed  their 
success  in  procuring  reliable  aircraft  equipment  to  the  RIW. 

So,  in  the  early  1970s  the  Air  Force  began  trial  applications  of 
the  RIW  in  selected  weapon  system  acquisitions. 

The  USAF  RIW  Pricing  Model,  then,  was  developed  to  evaluate 
this  one  specific  type  of  warranty,  the  RIW.  It  has  since  been 
used  to  establish  an  independent  warranty  price  estimation  prior 
to  the  receipt  of  contractor  bids  (15:33).  The  model  was  built 
on  a  bottom-up  accounting  methodology  (described  earlier)  which 
collects  costs  in  four  broad  categories:  Fixed  Direct  Costs, 
Other  Yearly  Costs,  Costs  per  Return,  and  Data  and 
Administration  costs  (15:3-3  to  3-4). 


The  advantages  inherent  in  this  model  are  those  associated 
with  any  bottom-up  accounting  methodology.  The  RIW  Pricing 
Model  offers  tremendous  potential  for  accurately  forecasting 
costs.  Also,  once  it  has  been  developed,  the  model  can  easily 
assess  ”...  the  effects  of  changing  system  performance 
parameters  and  warranty/guarantee  provisions"  (15:3-9). 
Flexibility  and  potentially  great  fidelity,  then,  are 
characteristics  of  the  USAF  RIW  Pricing  Model. 

Likewise,  the  RIW  Pricing  Model  cannot  escape  the 
disadvantages  associated  with  the  cost  estimating  methodology  at 
its  foundation.  These  stem  from  insufficient  historical  data 
and  the  risks  inherent  in  forecasting  future  costs, 
reliabilities,  and  repair  rates  beyond  the  initial  few  years  of 
a  system's  life  (15:3-6).  Such  forecasts  can  be  only  as  good  as 
the  supporting  historical  data  base  and  the  fidelity  of 
contractor  cost  estimates. 


Product  Performance  Agre ement  Center  Decision  Support  Sy stem 

The  Air  Force  Product  Performance  Agreement  Center  (PPAC), 
located  at  Wright-Patterson  AFB,  Ohio,  has  developed  a  computer- 
based  model  to  help  the  program  manager  better  cope  with  the 
intricacies  of  weapon  system  warranties.  The  PPAC  Decision 
Support  System  (DSS)  is  an  interactive  computer  program  that 
aids  the  program  manager  in  the  selection  of  system  parameters 
to  be  warranted,  the  optimization  of  warranty  type  and 
structure,  and  the  cost-benefit  analysis  of  warranties.  The 
cost-benefit  analysis  module  can  be  used  to  estimate  government 
warranty  costs,  contractor  costs,  and  the  expected  life  cycle 
costs  and  savings  inherent  in  the  warranty  (7:4-3). 

The  cost-benefit  analysis  module  requires  extensive  data  on 
predicted  weapon  system  performance,  support  system  performance, 
and  planned  system  operations.  For  example,  data  on  weapon 
system  performance  is  required  at  the  Line  Replaceable  Unit 
level  (LRU,  an  individual  component  within  an  aircraft 
subsystem)  in  order  to  use  the  model.  For  example,  the  user  of 
the  DSS  must  already  know  the  expected  number  of  failures  and 
removals  for  each  LRU,  as  well  as  the  number  of  manhours  that 
will  be  required  to  fix  it. 

This  requirement  for  extensive,  detailed  information 
severely  limits  the  application  of  this  model.  Such  data  is 
simply  not  available  as  early  as  the  source  selection  phase  of  a 
weapon  system  which  will  employ  new  technology,  or  current 
technology  integrated  in  a  new  way.  Also,  the  trend  in  weapon 
system  acquisition  is  toward  system-level  performance 
specifications  and  warranties.  Here,  the  government  is  less 
concerned  with  the  performance  of  each  individual  piece  of  a 


weapon  system.  It  is  the  overall  or  system-level  performance  of 
the  weapon  that  really  matters.  Under  such  a  system-level 
performance  specification,  the  contractor  is  left  with  the 
latitude  to  select  (within  certain  constraints)  whatever  mix  of 
LRUs  will  result  in  the  weapon  system  performing  at  specified 
values.  Consequently,  during  source  selection  it  is  unlikely 
that  either  the  government  or  the  contractor  will  know  the 
complete  complement  of  LRUs  that  will  ultimately  comprise  the 
whole  weapon  system.  As  a  result,  the  corresponding  LRU-level 
data  is  unknown  and  unknowable. 


Warranty  Analysis  Model  (WAM) 

The  Warranty  Analysis  Model  (WAM)  represents  an  important 
stride  in  the  direction  of  useful,  usable  warranty  evaluation 
tools.  WAM's  developers  recognized  the  application  of  a 
warranty  evaluation  tool  should  optimally  occur  during  the  early 
stages  of  the  weapon  system  acquisition  cycle  (e.g.,  source 
selection) ,  but  "...  will  require  implementation  without  the 
availability  of  detailed  equipment  and  operating  scenario 
characteristics"  (2:3).  As  discussed  earlier,  the  realization 
that  some  warranty  evaluation  models  require  information  that  is 
simply  not  available  during  the  source  selection  stage  is  an 
important  first  step  in  the  creation  of  a  useful  model. 
Accordingly,  WAM  incorporates  a  two  stage  evaluation  process. 
First,  a  qualitative  step  screens  out  sub-systems  and  equipment 
that  are  not  natural  candidates  for  warranty  protection.  This 
initial  screen  compares  a  candidate's  performance 
characteristics  to  a  "  .  .  .  list  of  warranty  evaluation 
criteria,"  in  order  to  yield  "...  a  qualitative 
determination  of  suitability"  (2:5). 

The  second  stage  of  WAM  subjects  the  surviving  warranty 
candidates  to  a  parametric  economic  analysis  to  determine  "  .  . 

.  the  incremental  differences  in  expected  life  cycle  acquisition 
and  operating  and  support  costs  due  to  incorporation  of  the 
[warranty]  program"  (2:1). 

As  with  the  USAF  RIW  Pricing  Model,  WAM  is  optimized  for  the 
analysis  of  one  specific  type  of  warranty,  the  RIW.  But  the  WAM 
modelers  have  allowed  for  the  tailoring  of  WAM  to  accommodate 
other  types  of  warranties  in  that  "...  the  methodology  can  be 
used  with  or  without  the  reliability  growth  module"  (2:2). 

WAM  is  packaged  in  the  form  of  an  interactive  computer 
program  and  requires  only  "...  standardized  input  data 
elements  which  are  available  early  in  any  product  design 
process"  (2:7).  The  model's  output  is  in  the  form  of  "  .  .  . 
detailed  listings  of  the  expected  acquisition,  warranty  and 
operating  and  support  costs"  for  a  range  of  warranty  periods  and 
life  cycles  (2:7) . 


These  models  typify  the  state  of  warranty  evaluation 
research.  The  bottom  line  is  there  exists  no  off-the-shelf 
methodology  a  program  manager  can  readily  apply  to  the  crucial 
determination  of  whether  or  not  a  proposed  weapon  system 
warranty  is  cost-beneficial.  Given  this  background,  the 
following  chapter  examines  the  nature  of  the  AC-130U  Gunship 
warranty  and  the  cost-benefit  analysis  methodology  developed 
for  its  evaluation. 


Chapter  Four 


AC-130U  WARRANTY  PROVISIONS 
AND 

COST-BENEFIT  ANALYSIS  MODEL 

The  following  describes  the  specific  warranty  provisions 
designed  into  the  AC-130U  contract.  These  provisions  were 
included  in  the  Request  for  Proposal  which  solicited  proposals 
from  many  contractors.  Contractors  were  required  to  price  each 
element  of  the  warranty  and  provide  their  pricing  rationale. 

The  Gunship  Program  Office  entered  formal  source  selection  with 
proposals  from  three  prime  contractors. 

The  latter  half  of  this  chapter  describes  the  cost-benefit 
model  which  was  developed  by  the  Gunship  Program  Office  and 
tailored  specifically  to  analyze  the  prime  contractors'  warranty 
price  proposals. 


AC-130U  WARRANTY  PROVISIONS 

The  planned  mission  of  the  AC-130U  system  and  the  projected 
cost  of  designing,  developing,  and  building  twelve  Gunships 
meant  the  program  was  bound  by  the  provisions  of  the  Defense 
Procurement  Reform  Act  (i.e.,  its  projected  cost  exceeded  both 
the  $100,000  per  unit  limit  and  the  $10  million  total 
procurement  cost  limit) .  The  Gunship  legally  required  a  cost- 
beneficial  warranty.  In  order  to  comply  with  the  Reform  Act, 
the  program  developed  provisions  that  corresponded  to  the  three 
required  warranties  referenced  in  Chapter  2:  Design  and 
Manufacturing,  Material  and  Workmanship,  and  Essential 
Performance  warranties.  Each  of  these  is  discussed  below. 


AC-130U  Design  and  Manufacturing  Warranty 

The  Gunship  Design  and  Manufacturing  Warranty  stipulates 
the  contractor  will  warrant: 

For  a  period  of  two  years  after  acceptance  of  the 
aircraft  or  a  period  of  two  years  after  completion  of 
Interim  Contractor  Support  .  .  .  whichever  is  later, 
that  [the  weapon  system]  will  conform  to  all  design  and 


manufacturing  requirements  specified  in  this  contract 
(including  but  not  limited  to  all  specifications  and 
statements  of  work) ,  and  in  any  amendments  thereto. 

Design  and  manufacturing  requirements  include,  but  are 
not  limited  to,  all  structural  and  engineering  plans  and 
manufacturing  particulars,  including,  but  not  limited 
to,  precise  measurements,  tolerance,  materials, 
processes  and  finished  product  tests  for  the  item  being 
produced  (9:2) . 

In  other  words,  this  provision  requires  the  contractor  to 
warrant  that  the  engineering  design  and  the  manufacturing 
processes  he  employs  to  produce  the  hardware  comply  with  the 
specifications  in  the  contract.  It  covers  characteristics  of 
the  weapon  system  such  as  "  .  .  .  size,  weight,  interfaces, 
power  requirements,  and  material  composition"  (3:2-3).  The 
Gunship  Program  Office  added  a  unique  feature  to  this  warranty 
in  its  description  of  the  warranty's  duration. 

The  Program  Office  recognized  the  necessity  and 
desirability  of  a  period  of  Interim  Contractor  Support  (ICS)  for 
the  Gunship.  During  this  time,  the  contractor  would  be 
responsible  for  maintaining  the  aircraft  and  making  any 
engineering  design  changes  required  to  maintain  compliance  with 
the  specification.  These  design  changes  might  entail  replacing 
Gunship  components  or  subsystems,  which  would  likewise  result  in 
changes  to  the  associated  support  and  test  equipment,  training 
and  training  equipment  requirements.  The  Air  Force  would  then 
use  this  interim  period  to  methodically  build-up  its  organic 
support  capability  for  the  Gunship  only  as  the  design  of  the 
weapon  system  matured  and  stabilized.  The  objective  in  keeping 
this  warranty  in  effect  for  two  years  beyond  the  end  of  the  ICS 
period  was  to  ensure  that  during  ICS  the  contractor  would  make 
design  changes  that  truly  addressed  the  cause  of  performance 
shortcomings.  From  the  Program  Office  perspective,  a  contractor 
would  be  more  likely  to  institute  quality,  lasting  fixes  knowing 
his  responsibility  for  system  performance  continued  well  into 
the  future. 


Material  and  Workmanship  Warranty 

The  Gunship  Material  and  Workmanship  Warranty  requires  the 
contractor  to  warrant: 

For  a  period  of  one  year  after  acceptance  of  the 
aircraft  or  a  period  of  one  year  after  completion  of 
Interim  Contractor  Support  .  .  .  whichever  is  later, 
that  the  [weapon  system]  .  .  .  [is]  free  from  all 
defects  in  materials  and  workmanship  (9:2). 

As  in  the  case  of  the  Design  and  Manufacturing  Warranty,  this 


clause  is  "standard"  with  the  exception  of  the  innovation  on  the 
part  of  the  Program  Office  to  establish  the  link  to  the  ICS 
period.  This  warranty  targets  a  system's  latent  defects,  or 
those  defects  that  "...  exist  at  the  time  of  acceptance  by 
the  Government  but  [do]  not  manifest  [themselves]  until 
sometime  after  acceptance"  (3:2-1,  2-5). 


Essentia 1  Performance  Warran ty 

The  warranty  that  proved  the  most  challenging  in 
development,  and  required  the  most  innovation  and  foresight,  was 
the  Essential  Performance  Warranty.  A  system's  essential 
performance  characteristics  are  those  operational  aspects 
required  of  the  system  in  order  for  it  to  successfully  perform 
its  intended  mission.  A  significant  constraint  in  choosing 
essential  performance  characteristics  is  the  extent  to  which 
that  parameter  is  already  measured  in  the  field.  The  selection 
of  a  characteristic  which  is  routinely  and  accurately  measured 
and  reported  by  the  user  is  desirable  and  will  help  to  minimize 
the  cost  (both  in  terms  of  dollars  and  inconvenience)  associated 
with  monitoring  the  chosen  parameter. 

Accordingly,  the  Gunship  Essential  Performance  Warranty 
requires  the  contractor  to  warrant  (9:2): 

For  a  period  of  two  years  after  acceptance  of  the 
aircraft  or  a  period  of  two  years  after  completion  of 
the  Interim  Contractor  Support  .  .  .  whichever  is  later, 
that  the  [weapon  system]  will  conform  to  the  essential 
performance  requirements  .  .  .  specifically  delineated 
in  this  contract  and  in  any  amendments  thereto.  For  the 
purposes  of  this  warranty,  the  essential  performance 
requirements  are  delineated  as  follows: 

Navigation  System  Accuracy 

(A)  Inertial  Navigation  System:  0.8  NM/hr  Circular 
Error  Probability  (CEP)  for  first  hour  without  position 
updates;  0.65  NM/hr  without  position  updates  from  one  to 
ten  hours . 

(B)  Global  Positioning  System:  15  meters  CEP  and  20 
meters  Spherical  Error  Probability. 

Fire  Control:  [Note:  Classified;  not  included  in  this 
paper .  ] 

Reliability:  1.1  Flight  Hours,  Mean  Flight  Hours 

Between  Unscheduled  Maintenance  (MFHBUMA) . 


Maintainability:  2.6  Hours  Mean  Time  to  Repair  ( MTTR ) . 


These  three  major  provisions,  then,  fulfilled  the  mandate  of  the 
Weapon  System  Warranty  Act  and  the  Defense  Procurement  Reform 
Act.  Coincidental  to  the  crafting  of  these  AC-130U  Gunship 
warranty  provisions,  the  Program  Office  also  developed  the 
following  warranty  evaluation  methodology. 


AC-130U  COST  BENEFIT  ANALYSIS  METHODOLOGY 


Purpose 

The  AC-130U  cost-benefit  analysis  model  was  developed  to 
answer  two  questions: 

1.  Were  the  proposed  warranties  cost  effective? 

2.  Which  proposal  was  the  most  beneficial  to  the  Air 

Force? 

In  answering  the  first  question  the  Program  Office  would  know 
the  status  of  ics  compliance  wi zh  DFARS  Subpart  46.770  and  the 
Defense  Procurement  Reform  Act.  Could  we  place  a  proposed 
warranty  on  contract  as  is,  or  would  it  require  modification  to 
make  the  warranty  cost  beneficial  to  the  government? 

The  second  question,  which  assumed  two  or  more  cost 
beneficial  warranty  proposals,  would  provide  additional  cost 
data  on  which  to  base  the  selection  of  the  winning  contract. 

So,  the  cost-benefit  analysis  served  the  source  selection  in  two 
valuable  ways.  First,  it  would  alert  the  source  selection 
committee  to  a  contract  unawardable  on  the  basis  of  a  non-cost 
beneficial  warranty.  Secondly,  the  cost-benefit  analysis  would 
serve  as  a  discriminator  between  competitive  contract  proposals. 


The  fundamental  methodology  upon  which  the  cost-benefit 
analysis  model  was  built  was  a  comparison  of  the  life  cycle 
costs  the  government  could  expect  to  incur  in  owning  and 
operating  the  AC-130U  over  the  course  of  its  service  life  both 
with  and  without  the  proposed  warranty.  Specifically,  the  AC- 
130U  Cost  Benefit  Analysis  sought  to  compare  the  cost  of  each 
offeror's  warranty  with  the  marginal  costs  the  Air  Force  would 
incur  without  a  weapon  system  warranty.  Our  original  intent  was 
to  employ  a  bottom-up  accounting  methodology  to  aggregate  the 
cost  elements  involved.  Consequently,  the  major  effort 
associated  with  the  development  of  the  model  was  determining  the 
most  accurate  source  or  basis  for  estimating  each  category  of 
cost  that  would  comprise  the  total  life  cycle  cost  of  the  AC- 
130U . 


While  some  categories  of  costs  were  well  documented  and 
could  be  accurately  estimated,  others  were  more  elusive,  their 
estimates  mere  educated  guesses.  For  example,  the  fuel 
consumption  of  the  C-130H  (the  basic  airframe  on  which  the  AC- 
130U  would  be  built)  is  known,  and  the  cost  of  JP4  is  relatively 
stable  (8:8,  19,  22).  So,  barring  some  future  energy  crisis, 
the  cost  of  fuel  over  the  life  of  the  Gunship  is  easily  and 
accurately  estimated.  On  the  other  hand,  what  is  the  cost  of 
missing  an  intended  target?  It  is  certainly  more  chan  the  cost 
of  the  expended  round.  An  almost  infinite  and  largely 
unquantif iable  litany  of  indirect  costs  comes  to  mind: 

Opportunity  Cost:  By  "wasting"  the  round,  what  other 
targets  go  unscathed,  and  what  would  have  been  the  value  in 
neutralizing  them? 

Needless  Exposure  to  Enemy  Threat:  Does  the  "miss"  result 
in  additional  time  over  target  and  increased  exposure  to  around 
fire? 

Unintended  Collateral  Damage:  What  did  the  round  actually 
hit?  Did  we  injure  or  kill  friendly  forces,  or  damage  their 
equipment? 

Continued  Effectiveness  of  Intended  Target:  The  target  is 
still  operational,  possibly  inflicting  damage  to  friendly 
forces . 

It  soon  became  apparent  that  certain  categories  and  types  of 
costs  would  evade  our  best  attempts  at  forecasting  them 
accurately.  Indeed,  one  rapidly  approaches  a  point  of 
diminishing  return  in  pursuit  of  the  "perfect"  cost  estimate. 

So,  through  a  rough  trial  and  error  process  we  accepted 
certain  costs  as  relevant  and  relatively  easily  and  accurately 
estimable.  At  the  same  time  other  costs  were  rejected  either 
because  they  were  not  relevant,  or  though  relevant,  were  too 
hard  to  accurately  forecast. 


Ground  Rules  and  Assumptions 

Throughout  this  trial  and  error  process,  we  were  forced  to 
establish  a  set  of  ground  rules  and  make  certain  assumptions  in 
order  to  make  the  CBA  manageable  given  our  limited  manpower  and 
time.  These  assumptions  allowed  us  to  "bound"  the  problem,  and 
are  illustrative  of  the  obstacles  encountered  during  the  CBA. 

a.  Warranty  Period.  Though  not  specifically  addressed  in 
the  contract,  we  assumed  the  warranty  would  begin  with  the  Air 
Force  acceptance  of  the  first  aircraft.  The  warranty  would 


then  remain  in  effect  through  the  ICS  period  and  end  six  months, 
one  year,  or  two  years  after  the  termination  of  depot  level  ICS 
(all  three  scenarios  were  examined  as  part  of  a  sensitivity 
analysis).  The  end  of  ICS  was  set  at  the  end  of  CY93  for  all 
offerors,  regardless  of  individual  delivery  and  ICS  schedules, 
although  individual  delivery  schedules  provided  the  number  of 
aircraft  in  the  warranty  period  (9:9). 

b.  Post-Warranty  Period .  We  assumed  all  corrective  actions 
initiated  under  the  warranty  would  be  completed  prior  to  the 
expiration  of  the  warranty.  Furthermore,  the  aircraft  were 
assumed  to  operate  within  specification  tolerances  up  to  the 
warranty  expiration  (9:9). 

c.  Deficiency  Remedies .  We  assumed  any  deficiency  covered 
by  the  warranty  (e.g..  Design  and  Manufacturing)  would  be 
corrected  by  means  of  a  Class  4  modification  in  both  the  "with" 
and  "without"  warranty  cases.  (A  Class  4  modification  is 
implemented  to  achieve  acceptable  levels  of  safety,  overcome 
mission  capability  deficiencies,  improve  reliability  or  reduce 
maintenance  costs,  all  deficiencies  that  our  warranty  was 
designed  to  cover  (8:3).  The  rationale  for  this  assumption  is 
central  to  the  cost-benefit  analysis.  The  alternative  to  the 
permanent  "fix"  of  a  Class  4  modification,  some  kind  of 
temporary,  work-around  measure,  invariably  results  in  increased 
logistics  support  of  the  ailing  system.  In  other  words,  instead 
of  correcting  the  cause  of  the  deficiency,  the  logistics  system 
provides  a  "band  aid"  in  the  form  of  increased  stockage  of  spare 
parts,  providing  additional  maintenance  man-hours,  cannibalizing 
good  systems,  hiring  full-time  contractor  "tech  reps",  or 
various  other  means.  Over  the  life  cycle  of  a  weapon  system, 
the  cost  of  such  work-arounds  can  be  enormous.  However,  an 
assumption  basic  to  the  cost-benefit  analysis  was  the  Air  Force 
would  insist  system  performance  remain  within  specified 
contractual  values  whether  a  warranty  was  purchased  or  not. 
Consequently,  the  Air  Force  would  opt  for  a  Class  4  modification 
to  correct  a  deficiency  with  or  without  a  warranty  on  Gunship 
system  performance  (9:9). 

d.  Flyaway  Costs .  The  estimated  flyaway  cost  for  each 
proposal  excluded  the  cost  of  the  government  furnished  equipment 
( GFE )  (that  equipment  the  government  had  agreed  to  make 
available  to  the  contractor  as  part  of  his  system  integration 
effort).  Consequently,  the  cost  of  the  entire  C-130H  airframe, 
which  the  Air  Force  would  provide  the  winning  contractor  as  GFE, 
was  not  included  in  the  forecast  flyaway  cost  (9:10). 

e.  Government  Warranty  Administration  Costs .  These  are 
direct  and  indirect  costs  the  Air  Force  incurs  in  managing  the 
warranty,  i.e.,  ensuring  and  documenting  compliance  with  the 
terms  and  conditions  of  the  warranty.  Typical  warranty 
administration  costs  include  system  performance  tracking  and 


reporting,  maintenance  documentation  and  review,  vendor  contact, 
and  preparing  and  reviewing  status  reports.  In  the  case  of  the 
Gunship,  we  deliberately  structured  the  warranty  to  minimize  the 
costs  of  its  administration.  For  example,  one  criteria  for  the 
selection  of  Essential  Performance  Characteristics  was  the 
performance  parameter  chosen  would  not  require  the  imposition  of 
any  reporting  procedure  over  and  above  normal  Air  Force 
maintenance  reporting  procedures  (5; 6; 9: 10). 


Scope 

Initial  iterations  of  our  cost-benefit  analysis  model  were 
considerably  more  ambitious  and  comprehensive  than  the  version 
we  finally  applied.  At  the  start,  we  intended  to  capture  all 
relevant  costs  that  comprise  a  weapon  system's  life  cycle  cost 
and  calculate  the  impact  (savings)  a  comprehensive  system 
warranty  would  have.  It  became  readily  apparent  that  a  project 
of  that  magnitude  was  beyond  our  capability  and  time  allowed. 
After  all,  this  cost-benefit  analysis  effort  was  an  ancillary 
duty  of  a  small  group  of  program  management  and  functional 
personnel  whose  main  task  was  the  conduct  and  management  of  a 
major  weapon  system  source  selection.  So,  the  final  version  of 
the  cost-benefit  analysis  model  reflected  a  series  of 
suboptimizations  based  on  our  limited  pool  of  manpower,  lack  of 
cost-benefit  analysis  expertise,  and  the  short  time  involved. 

We  finally  settled  on  a  version  of  the  model  that  captured 
only  the  cost  of  the  Class  4  modifications  the  Gunship  would 
likely  require  to  maintain  specified  contractual  performance 
during  the  early  years  in  the  field.  AFR  173-13  provided  an 
accepted  algorithm  for  estimating  Class  4  modification  costs 
(8:128).  That  algorithm  coupled  with  specified  and  contractor- 
advertised  system  reliability  parameters  provided  us  a  "neat," 
if  narrow  view  of  the  marginal  value  of  a  proposed  warranty. 

In  sum,  our  cost-benefit  analysis  methodology  compared 
total  Class  4  modification  costs  to  the  price  of  the  proposed 
warranty  for  alternative  warranty  period  lengths  to  isolate  the 
marginal  costs.  The  marginal  costs  would  indicate  whether  a 
proposed  warranty  was  cost-effective  or  not. 


Application 

The  Gunship  warranty  cost-benefit  analysis  was  conducted 
during  the  formal  source  selection  process,  the  purpose  of  which 
was  to  award  a  contract  for  the  Full  Scale  Development  and 
Production  of  the  AC-130U  Gunship  to  one  of  three  competing 
contractors.  The  Program  Office,  and  other  functional  personnel 
conducting  the  source  selection,  performed  a  separate  series  of 
analyses  on  each  competing  warranty  bid.  This  was  necessary 


since  each  contractor  generated  a  unique  set  of  input  data  based 
on  his  estimates  of  such  factors  as  system  and  sub-system 
reliability,  frequency  of  repair,  cost  per  repair,  cost  for 
spares,  training,  and  technical  manuals,  to  mention  only  a  few. 
The  resulting  estimated  warranty  cost  for  each  contractor  was 
compared  to  the  Air  Force's  independent  estimate  of  the  cost  of 
providing  organic  maintenance.  The  difference  was  then  a 
measure  of  each  warranty's  cost-benef iciality .  The  estimated 
warranty  cost  was  also  included  in  the  overall  forecast  life 
cycle  cost  of  each  contractor's  weapon  system.  Consequently, 
the  warranty  cost-benefit  analysis  counted  heavily  in  the  source 
selection . 

The  Cost  Panel  of  the  source  selection  organization  worked 
independently  to  derive  the  bottom  line  life  cycle  cost  for  each 
contractor's  bid.  As  a  rule,  no  other  functional  sub-set  of  the 
source  selection  activity  is  granted  access  to  the  cost  data. 
However,  the  Cost  Panel  relied  heavily  on  technical  advice  and 
inputs  from  the  groups  performing  the  technical  evaluation  of 
the  bids--engineers ,  logisticians,  safety  people,  and 
operators.  In  this  way  the  costs  ultimately  assigned  to  the 
operation  and  support  of  the  Gunship  received  technical 
validation . 

The  cost-benefit  analysis  relied  heavily  on  data  requested 
of  each  contractor  as  part  of  his  proposal.  Various  factors 
worked  against  the  timely  transmission  of  this  vital 
information.  (See  Chapter  5)  As  a  result,  in-house  cost 
estimates  were  substituted  for  contractor  inputs  in  a  few 
instances . 

The  warranty  analysis  resulted  in  the  determination  that 
one  contractor's  warranty  was  cost-beneficial  to  the  Air  Force, 
while  the  others  were  not.  Another  product  of  the  warranty 
evaluation  effort  was  a  series  of  sensitivity  analyses  that 
provided  the  Source  Selection  Authority  the  relative  costs  and 
benefits  associated  with  each  warranty  for  various  lengths  of 
time.  The  objective  here  was  to  provide  the  final  decision¬ 
maker  the  necessary  information  to  optimize  the  contracted 
warranty . 

In  sum,  the  three  main  AC-130U  warranty  provisions  covered 
Design  and  Manufacturing,  Material  and  Workmanship,  and 
Essential  Performance  Requirements,  as  required  by  the  Weapon 
System  Warranty  Act  and  the  Defense  Procurement  Reform  Act.  Of 
the  three,  the  Essential  Performance  Requirements  provision 
required  the  greatest  vision  and  innovation.  Also,  the  AC-130U 
Gunship  warranty  cost-benefit  analysis  methodology  complied 
with  the  intent  of  the  two  enactments.  While  the  cost-benefit 
analysis  was  a  pioneering  effort,  as  confirmed  by  the  GAO  report 
cited  in  Chapter  2,  it  exhibited  certain  shortcomings  discussed 
in  the  next  chapter. 


Chapter  Five 


ANALYSIS  OF  AC-130U  WARRANTY  COST-BENEFIT  ANALYSIS 

Obviously  there  were  faults  in  the  Gunship  cost-benefit 
analysis.  Clearly  there  were  costs  and  savings  associated  with 
the  weapon  system  warranty  other  than  those  we  captured.  But 
the  fact  that  the  AC-130U  Gunship  Program  Office  made  an  honest 
attempt  to  conduct  the  cost-benefit  analysis  at  all  deserves 
recognition.  After  all,  the  GAO  report  cited  in  Chapter  Two 
indicated  that  AC-130U  warranty  analysis  effort  was  indeed 
exceptional.  Nevertheless,  the  following  analysis  of  the 
strengths  and  weaknesses  inherent  in  our  cost-benefit  analysis 
is  offered  in  the  hope  that  future  analyses  are  even  more 
comprehensive  and  accurate. 


STRENGTHS 

The  underlying  strength  of  the  Gunship  cost-benefit  analysis 
was  its  grounding  in  Life  Cycle  Cost  (LCC)  theory  and  analysis. 
The  methodology  was  developed  and  the  analyses  were  performed  by 
LCC  analysis  experts. 

Another  significant  strength  of  the  cost-benefit  analysis 
effort  was  the  validation  applied  to  the  cost  estimates  from  the 
functional  experts.  As  the  model  was  developed,  logistics, 
engineering,  and  operational  experts  were  interviewed  in  an 
attempt  to  validate  the  performance,  support,  and  operational 
scenarios  captured  in  the  model,  and  in  an  attempt  at 
translating  performance  parameters  into  costs  and  benefits 
measurable  in  dollars. 

Finally,  the  foresight  that  characterized  the  development  of 
the  basic  warranty  provisions  as  well  as  the  warranty  cost- 
benefit  analysis  methodology  was  a  noteworthy  attribute.  The 
Program  Office  recognized  the  two  processes  had  to  occur 
simultaneously.  In  the  case  of  the  essential  performance 
requirements  provision,  the  weapon  system  attributes  identified 
as  essential  would  influence  the  evaluation  scheme,  which  would, 
in  turn,  dictate  the  information  necessary  to  conduct  the  cost- 
benefit  analysis. 

The  strengths  of  the  AC-130U  warranty  cost-benefit  analysis 


methodology  were  indeed  noteworthy.  Its  shortcomings  were 
perhaps  even  more  instructive. 


WEAKNESSES 

The  fundamental  shortcoming  of  the  Gunship  warranty  cost- 
benefit  analysis  was  the  narrow  scope  of  its  final  version. 
Ideally,  the  analysis  would  have  embraced  many  more  of  the  costs 
associated  with  operating  and  supporting  the  weapon  system 
covered  by  the  three  warranties:  Design  and  Manufacturing, 
Material  and  Workmanship,  and  Essential  Performance 
Requirements.  However,  ultimately  we  were  constrained  to  an 
over-simplified  comparison  of  Class  4  modification  costs  because 
of  our  inability  to  quantify  and  forecast  such  costs  as: 

1)  The  cost  of  the  system  not  firing  accurately.  What 
are  the  costs  of  missing  the  target?  Certainly  more 
than  just  the  cost  of  another  round.  Other  costs  that 
would  have  to  be  considered  and  quantified  include: 
increased  risk  to  the  Gunship  due  to  extended  time-over¬ 
target;  additional  wear  and  tear  on  the  weapon  system; 
risk  to  friendly  ground  forces  stemming  from  failure  to 
destroy  intended  target  with  first  shot;  etc.  ad 
infinitum. 

2)  The  cost  of  an  inaccurate  navigation  system.  What 
are  the  costs  of  failing  to  position  the  Gunship  along 
the  intended  route?  The  risk  of  over-flying  hostile 
anti-aircraft  weapon  positions;  the  risk  of  alerting 
the  enemy  as  to  the  Gunship 's  position  and  subsequently 
jeopardizing  the  mission;  reduction  in  time-over-target 
resulting  from  inefficient  enroute  navigation.  Again, 
the  possibilities  are  almost  endless,  and  the  decision 
as  to  which  should  be  included  is  as  difficult  as  the 
process  of  forecasting  and  quantifying  these  costs. 

3)  The  cost  of  the  system  performing  below  the  specified 
values  of  reliability.  Here  again,  the  possible  costs 
to  be  considered  are  seemingly  infinite.  For  example, 
the  costs  of  a  system  or  subsystem  malfunctioning  or 
failing  sooner  than  expected  might  include  the  cost  of 
subsequently  achieving  the  desired  reliability  by  adding 
redundant  systems,  the  risk  of  not  achieving  the 
intended  mission  due  to  equipment  failure,  the  risk  of 
losing  the  weapon  system  resulting  from  the  failure  of 
critical  equipment. 

4)  The  cost  of  the  system  performing  below  the  specified 
values  of  maintainability.  Examples  here  include:  the 
cost  of  additional  unplanned  maintenance  manhours 
required  to  maintain  the  system;  the  cost  of  increased 


down  time  for  the  Gunship;  the  cost  of  additional 

special  test  equipment  required  to  support  the  system. 

These  examples  illustrate  the  monumental  task  of  "costing"  just 
one  of  the  three  Gunship  warranties,  the  Essential  Performance 
Requirements  warranty,  and  explain  why  we  opted  to  estimate 
these  costs  with  a  surrogate  measure.  Certainly  the  surrogate 
we  chose,  the  forecast  cost  of  Class  4  modifications,  would 
capture  pieces  of  each  of  these  costs.  But  the  resulting  cost 
estimate  was  at  best  a  rough,  order  of  magnitude  measure. 

Another  difficulty  with  the  cost-benefit  analysis  was  its 
dependence  on  data  that  ultimately  was  unattainable  or  non¬ 
existent.  One  important  intended  source  of  data  was  the 
contractors  bidding  on  the  Gunship  contract.  The  government's 
Request  for  Proposal  and  Instructions  to  Offerors  (documents 
used  to  solicit  contractor  bids,  and  to  describe  the  format  and 
types  of  information  required  in  the  contractors'  bids, 
respectively)  required  the  contractor  to  separately  break-out 
the  costs  associated  with  each  piece  of  his  warranty.  In 
addition,  the  government  requested  a  description  of  the 
methodology  employed  to  generate  the  contractors'  warranty  cost 
estimates . 

The  responses  to  these  data  requests  were  dismal.  In  fact, 
repeated  requests  for  this  information  throughout  the  source 
selection  process  yielded  data  of  only  marginal  usefulness. 
During  face-to-face  negotiations  with  each  competing  contractor 
midway  through  source  selection,  the  explanation  for  this  lack 
of  responsiveness  became  apparent.  One  contributing  factor 
stemmed  from  the  contractors'  misunderstanding  of  the  purpose 
and  requirements  of  the  Gunship  warranty.  But  more 
significantly,  like  the  government,  the  contractors  were  unable 
to  accurately  forecast  the  cost  elements  associated  with  the 
type  of  warranties  the  government  had  crafted.  The  contractors 
had  nothing  that  remotely  resembled  a  rigorous  methodology  for 
estimating  the  warranty  costs.  The  cost  estimates  these  major 
defense  contractors  ultimately  provided  were  gross 
approximations  based  more  on  intuition  than  analysis. 

Also  conspicuous  in  its  absence  was  the  necessary  historical 
data  on  the  government  side.  For  example,  in  an  attempt  to 
verify  the  cost  factors  we  applied  in  our  forecast  of  Class  4 
modifications  over  the  life  of  the  Gunship,  we  attempted  to 
locate  historical  records  of  the  Class  4  modification  history  of 
other  similar  systems.  Unfortunately  we  discovered  that  the  Air 
Force  does  not  track  such  data.  Without  such  historical  data, 
estimating  future  performance  and  costs  is  a  difficult  task. 

A  third  weakness  inherent  in  the  Gunship  cost-benefit 
analysis  was  the  set  of  assumptions  on  which  it  was  ultimately 
based.  Due  to  the  paucity  of  analytical  tools  and  historical 


data  available,  we  were  forced  to  bound  the  Gunship  cost-benefit 
analysis  methodology  with  certain  unrealistic  assumptions.  One 
assumption  with  enormous  potential  for  invalidating  the 
government's  warranty  cost  estimates  was  related  to  our  linking 
of  the  ICS  phase  with  the  start  of  the  warranty  period.  The 
Program  Of f ice • hypothesized  that  the  contractor  would  less 
likely  apply  patchwork,  stopgap  fixes  to  performance 
specification  deviations  encountered  during  ICS  knowing  that  his 
responsibility  for  system  performance  extended  well  beyond  the 
ICS  phase. 

Such  thinking  may  prove  optimistic;  the  assumed  benefits 
have  never  been  proved.  Conceivably,  the  linking  of  ICS  and  the 
warranty  period  may  have  very  little  effect  on  the  quality  of 
engineering  changes.  Accordingly,  the  Gunship  cost-benefit 
analysis  may  have  over-estimated  the  true  value  of  contractor 
engineering  changes.  Other  assumptions  like  this  one  may  have 
unduly  constrained  the  Gunship  cost-benefit  analysis  effort,  but 
were  necessary  to  compensate  for  our  inability  to  simulate  and 
forecast  so  many  unknowns. 

Also,  the  assumption  that  during  ICS  and  the  subsequent 
warranty  period  the  contractor  would  be  inclined  to  implement 
Class  4  modifications  instead  of  temporary  work-arounds  may 
ultimately  prove  overly-optimistic  and  naive.  The  potential 
ramifications  in  this  scenario  are  significant.  The  impact  on 
the  AC-130U  Gunship  logistics  support  system  in  terms  of 
increased  down-time  and  turn-around  time,  increased  spare  parts 
requirements,  etc.,  would  quickly  outstrip  any  expected  warranty 
benefits . 

In  sum,  the  AC-130U  Gunship  warranty  cost-benefit  analysis 
effort  was  a  good,  honest,  pioneering  effort.  Clearly  though 
there  is  considerable  room  for  improvement.  The  final  chapter 
draws  conclusions  and  offers  recommendations  for  improving  this 
crucial  process. 


Chapter  Six 


CONCLUSIONS  AND  RECOMMENDATION 


CONCLUSIONS 

It  is  now  a  legal  requirement  for. DoD  acquisition  agencies 
to  contract  for  cost-beneficial  warranties  on  the  performance  of 
the  weapon  systems  they  procure.  Consequently,  cost-benefit 
analyses  must  be  performed  in  order  to  ensure  the  government 
procures  only  cost-effective  warranties.  However,  warranty 
cost-benefit  analysis  is  a  new  art.  There  exists  no  extensive 
body  of  knowledge  on  how  to  conduct  an  effective  warranty  ccst- 
benefit  analysis.  In  spite  of  the  enormous  impact  this  crucial 
decision-making  process  can  have  on  the  efficiency  and 
effectiveness  of  the  nation's  weapon  systems,  there  are  very  few 
useful  warranty  evaluation  tools  available  today. 

Some  warranty  evaluation  tools  are  beginning  to  emerge.  For 
reasons  explored  earlier,  none  of  these  are  suitable  for  use  in 
a  source  selection  environment.  The  ramifications  are 
potentially  serious.  The  assertion  has  been  made  that  "... 
the  actual  implementation  of  warranty  programs  has  been  held 
back  .  .  .  [by]  .  .  .  the  lack  of  appropriate  existing 
analytical  tools"  (2:3).  The  longer  the  government  fails  to 
contract  for  effective  weapon  system  warranties,  the  longer  we 
face  the  prospect  of  acquiring  weapon  systems  that  perform  at  a 
level  below  their  potential. 

A  significant  inadequacy  of  the  existing  models  is  their 
requirement  for  information  about  the  performance 
characteristics  of  a  weapon  system  far  in  excess  of  what  is 
realistically  available  or  which  can  be  accurately  forecast. 

This  is  especially  true  for  those  methodologies  that  build  upon 
a  bottom-up  accounting  approach  to  cost  estimation.  For  a 
weapon  system  that  exploits  new  technology,  or  one  that 
integrates  existing  technology  in  a  new  way,  or  for  a  contract 
written  on  a  system-level  performance  specification,  that  data 
is  simply  not  available  in  the  source  selection  phase  of  a 
program. 

A  second  reason  why  existing  models  do  not  lend  themselves 
to  use  during  the  source  selection  process  is  the  effective  use 


of  these  models  requires  extensive  cost-benefit  analysis 
expertise  or  expertise  in  the  manipulation  of  rigorous 
mathematical  models.  As  sophisticated  as  the  typical  Air  Force 
program  office  is,  that  kind  of  expertise  is  not  likely  to 
reside  there. 

Third,  these  models  seldom  enjoy  universal  applicability. 
Each  has  been  developed  to  analyze  a  specific  type  of  warranty. 
Consequently,  they  cannot  be  readily  tailored  for  use  with  other 
forms  of  warranties . 

The  AC-130U  Gunship  Program  Office  encountered  each  of  these 
obstacles  in  the  development  and  implementation  of  its  warranty 
cost-benefit  analysis  methodology.  That  methodology  was  the  end 
product  of  a  series  of  compromises  and  suboptimizations  stemming 
from  these  roadblocks.  Consequently,  it  was  clearly  not  an 
optimal  warranty  evaluation  scheme. 


RECOMMENDATION 

The  DoD  must  provide  the  impetus  for  the  design  and 
development  of  useful  warranty  analysis  tools.  The  type  of  tool 
most  useful  to  a  program  manager  in  the  source  selection 
environment  would  be  simple  and  flexible,  and  should  not  require 
extensive  data  on  the  future  performance  of  new  systems  for 
which  no  historical  data  base  exists.  The  warranty  cost- 
benefit  analysis  tool  should  not  require  great  expertise  in 
cost-benefit  analysis,  mathematical  modeling  or  computer 
science.  It  should  be  designed  for  the  typical  program  manager 
or  his  Deputy  Program  Manager  for  Logistics. 

The  development  of  useful  and  accurate  warranty  cost-benefit 
analysis  tools  will  require  considerable  dedication  and  effort. 
But  considering  the  importance  of  the  decision  at  hand,  and  the 
crucial  missions  of  the  weapons  in  question,  there  is  no  doubt 
the  effort  must  be  made. 
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